Image display device, image display system, image display method, and program

ABSTRACT

An image display device includes an approach situation information acquirer that acquires approach situation information indicating approach of at least part of a customer&#39;s body to a shelf on which a plurality of items are displayed and indicating, when there is the approach, a position of the approach on the shelf at each time; an associator that associates the approach situation information with one item of the plurality of items according to the position indicated by the approach situation information; an index value calculator that calculates an index value relating to the customer&#39;s behavior on the basis of information indicating the item associated with the approach situation information; and a display unit that displays the index value calculated by the index value calculator together with an image of the item on the basis of the association made by the associator.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/085,638, filed Sep. 17, 2018, which is a National Stage ofInternational Application No. PCT/JP2017/009533, filed Mar. 9, 2017,claiming priority based on Japanese Patent Application No. 2016-057537,filed Mar. 22, 2016, the contents of all of which are incorporatedherein by reference in their entirety.

TECHNICAL FIELD

The present invention relates to an image display device, an imagedisplay system, an image display method, and a program.

BACKGROUND ART

In some cases, customers' behavior in shops is analyzed for the purposeof determining the purchasing habits of customers or the like. Forexample, Patent Document 1 describes that a period of time during whicha customer stays in a surrounding area and the number of times of itempickup in each display shelf are measured/counted and displayed.

CITATION LIST Patent Literature

[Patent Document 1]

Japanese Unexamined Patent Application, First Publication No. 2015-11649

SUMMARY OF INVENTION Technical Problem

From the viewpoint of performing more sophisticated analysis ofcustomers' behavior, it is desirable that display be performed in moredetail than in the case of measuring/counting and displaying the periodof time during which a customer stays in a surrounding area and thenumber of times of item pickup in each display shelf.

An object of the present invention is to provide an image displaydevice, an image display system, an image display method, and a programthat can solve the above problems.

Solution to Problem

According to a first aspect of the present invention, an image displaydevice includes an approach situation information acquirer configured toacquire approach situation information indicating approach of at leastpart of a customer's body to a shelf on which a plurality of items aredisplayed and indicating, when there is the approach, a position of theapproach on the shelf at each time; an associator configured toassociate the approach situation information with one item of theplurality of items according to the position indicated by the approachsituation information; an index value calculator configured to calculatean index value relating to the customer's behavior on the basis ofinformation indicating the one item associated with the approachsituation information; and a display unit configured to display theindex value calculated by the index value calculator together with animage of the item on the basis of an association made by the associator.

According to a second aspect of the present invention, an image displaysystem includes a shelf-front behavior measurement sensor configured todetect approach of at least part of a customer's body to a shelf onwhich a plurality of items are displayed and to generate sensing dataindicating a position on the shelf at which the approach has beendetected; an approach situation information acquirer configured togenerate, on the basis of the sensing data, approach situationinformation indicating whether or not there is the approach andindicating, when there is the approach, the position on the shelf ateach time; an associator configured to associate the approach situationinformation with one item of the plurality of items according to theposition indicated by the approach situation information; an index valuecalculator configured to calculate an index value relating to thecustomer's behavior on the basis of information indicating the one itemassociated with the approach situation information; and a display unitconfigured to display the index value calculated by the index valuecalculator together with an image of the item on the basis of anassociation made by the associator.

According to a third aspect of the present invention, an image displaymethod includes an approach situation information acquisition step toacquire approach situation information indicating approach of at leastpart of a customer's body to a shelf on which a plurality of items aredisplayed and indicating, when there is the approach, a position of theapproach on the shelf at each time; an association step to associate theapproach situation information with one item of the plurality of itemsaccording to the position indicated by the approach situationinformation; an index value calculation step to calculate an index valuerelating to the customer's behavior on the basis of informationindicating the one item associated with the approach situationinformation; and a display step to display the index value calculated inthe index value calculation step together with an image of the item onthe basis of the association made by the association step.

According to a fourth aspect of the present invention, a program causesa computer to execute an approach situation information acquisition stepto acquire approach situation information indicating approach of atleast part of a customer's body to a shelf on which a plurality of itemsare displayed and indicating, when there is the approach, a position ofthe approach on the shelf at each time; an association step to associatethe approach situation information with one item of the plurality ofitems according to the position indicated by the approach situationinformation; an index value calculation step to calculate an index valuerelating to the customer's behavior on the basis of informationindicating the one item associated with the approach situationinformation; and a display step to display the index value calculated inthe index value calculation step together with an image of the item onthe basis of the association made by the association step.

Advantageous Effects of Invention

According to the present invention, it is possible to displayinformation indicating a customer's behavior in more detail.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic block diagram showing a functional configurationof an image display system according to an embodiment of the presentinvention.

FIG. 2 is an explanatory diagram showing an exemplary arrangement of ashelf-front behavior measurement sensor and a shelf situation imagingdevice when shelves are viewed from the side in the embodiment.

FIG. 3 is an explanatory diagram showing an exemplary arrangement ofshelf-front behavior measurement sensors when the shelves are viewedfrom the front in the embodiment.

FIG. 4 is an explanatory diagram showing an example of display of acustomer behavior index value by a display unit according to theembodiment.

FIG. 5 is an explanatory diagram showing an example of display, by thedisplay unit, of a customer behavior index value indicating acorrelation between a behavior that a customer has performed for an itemdesignated by a user of the image display system according to theembodiment and a behavior that the same customer has performed for anitem other than the designated item.

FIG. 6 is an explanatory diagram showing an exemplary data structure ofinformation stored in a storage unit according to the embodiment inwhich a sensing time and item identification information are associatedwith each other.

FIG. 7 is a flowchart showing an example of a processing procedure inwhich the image display device according to the embodiment generatesassociation information.

FIG. 8 is a flowchart showing a first example of a processing procedurein which the image display device according to the embodiment displays acustomer behavior index value.

FIG. 9 is a flowchart showing a second example of the processingprocedure in which the image display device according to the embodimentdisplays a customer behavior index value.

FIG. 10 is an explanatory diagram showing a minimal configuration of theimage display device according to the present invention.

FIG. 11 is an explanatory diagram showing a minimal configuration of theimage display system according to the present invention.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present invention will be described.However, the following embodiments do not limit the invention accordingto the claims. In addition, not all combinations of the featuresdescribed in the embodiments are necessarily essential to the solutionof the invention.

FIG. 1 is a schematic block diagram showing a functional configurationof an image display system according to an embodiment of the presentinvention. As shown in FIG. 1, the image display system 1 includes ashelf-front behavior measurement sensor 110, a shelf situation imagingdevice 120 and an image display device 200. The image display device 200includes a communicator 210, a display unit 220, an operation input unit230, a storage unit 280 and a controller 290. The controller 290includes an approach situation information acquirer 291, an associator292 and an index value calculator 293.

The image display system 1 analyzes a customer's shelf-front behaviorand displays the analysis result such that the analysis result issuperimposed on an image of shelves. The shelf-front referred to here isthe front of an item display shelves (in particular, the vicinity of thefront face of the item display shelves), and the shelf-front behaviorreferred to here is a behavior performed by the customer in front of theitem display shelves. Hereinafter, the item display shelf is simplyreferred to as a shelf.

The shelf-front behavior measurement sensor 110 is a sensor provided todetect shelf-front behaviors. For example, a distance sensor installedin a direction from the upper side to the lower side of the front faceof the shelves can be used as the shelf-front behavior measurementsensor 110. When a customer extends their hand to the shelves, theshelf-front behavior measurement sensor 110 detects horizontal andvertical positions of the customer's hand in front of the shelves,thereby allowing the image display device 200 to estimate an item towhich the customer has extended their hand.

The shelf situation imaging device 120 includes a camera that isprovided in front of the shelves to capture a moving image or a stillimage, and images the state of the front of the shelves and the state ofthe shelves continuously or at predetermined time intervals.

Using the image captured by the shelf situation imaging device 120, theimage display device 200 can detect a period from when the customerstops in front of the shelves to when leaving the shelves. As a result,the image display device 200 can detect hand extension that the customerperforms for the shelves from when the customer stops in front of theshelves to when leaving the shelves as a series of hand extensions ofthe same customer.

Further, by comparing images before and after the customer extends theirhand to the shelves, the image display device 200 can detect that thecustomer has picked up an item and that the customer has returned theitem picked up by the customer to the shelves.

Here, an exemplary arrangement of the shelf-front behavior measurementsensor 110 and the shelf situation imaging device 120 will be describedwith reference to FIGS. 2 and 3.

FIG. 2 is an explanatory diagram showing an exemplary arrangement of theshelf-front behavior measurement sensor 110 and the shelf situationimaging device 120 when the shelf is viewed from the side thereof. Inthe example shown in FIG. 2, the shelf-front behavior measurement sensor110 is installed downward from a ceiling 930 above the front face of theshelves 910. The shelves 910 has a plurality of layers and the items 920are placed on each of the layers. When the customer extends their handto the shelves 910, the shelf-front behavior measurement sensor 110measures the distance between the shelf-front behavior measurementsensor 110 and the hand. Using the measured distance, the image displaydevice 200 can determine which layer of the shelves 910 the customer hasextended their hand to.

The camera of the shelf situation imaging device 120 is installed at aposition and orientation for imaging the shelves 910 from the ceiling930 in front of the shelves 910. For example, the shelf situationimaging device 120 includes one camera, and the camera is installed at aposition and orientation at which it is possible to image the entirefront face of the shelves 910. Alternatively, the shelf situationimaging device 120 may include a plurality of cameras when the entirefront face of the shelves 910 cannot be imaged by one camera.

FIG. 3 is an explanatory diagram showing an exemplary arrangement ofshelf-front behavior measurement sensors 110 when the shelf is viewedfrom the front. In the example of FIG. 3, a plurality of shelf-frontbehavior measurement sensors 110 are arranged on the ceiling 930 alongthe width direction of the shelves 910. When a customer extends theirhand to the shelves 910, the image display device 200 can determine theposition of the hand in the horizontal direction of the shelves 910 byidentifying a shelf-front behavior measurement sensor 110 that hasdetected the hand.

Here, the shelf-front behavior measurement sensors 110 only need to beable to detect the position of the hand in the vertical direction of theshelves 910 and the position of the hand in the horizontal direction ofthe shelves 910 when the customer extends their hand to the shelves 910.For example, as the shelf-front behavior measurement sensors 110, oneset of three-dimensional cameras may be installed above the front faceof the shelves 910 to image the front face of the shelves in thevertical direction.

The installation position of the shelf-front behavior measurementsensors 110 is not limited to above the shelves 910. For example, theshelf-front behavior measurement sensor 110 may be installed on the sideof the shelves 910 to sense the front side of the shelves 910 sideways.

The image display device 200 analyzes a shelf-front behavior on thebasis of sensing data from the shelf-front behavior measurementsensor(s) 110.

Specifically, as described with reference to FIGS. 2 and 3, when thecustomer extends their hand to the shelves 910, the image display device200 detects the position of the hand in the vertical direction of theshelves 910 and the position of the hand in the horizontal direction ofthe shelves 910 on the basis of the sensing data from the shelf-frontbehavior measurement sensor 110. The image display device 200 stores thearrangement of the items 920 on the shelves 910 in advance and estimatesan item to which the customer has extended the hand on the basis of theposition of the hand of the customer. For example, for each item, theimage display device 200 counts the number of times that customers haveextended their hands to the item and displays the count result (forexample, a total count for all customers within a predetermined period)such that the count result is superimposed on the image of the shelves910.

The image display device 200 is configured, for example, using acomputer.

The communicator 210 communicates with other devices. The communicator210 receives sensing data from the shelf-front behavior measurementsensor 110. The communicator 210 also receives the image captured by theshelf situation imaging device 120 as image data.

For example, the display unit 220 has a display screen such as a liquidcrystal panel or a light emitting diode (LED) panel and displays variousimages under the control of the controller 290. The display unit 220displays a customer behavior index value such as the count result of thenumber of times that customers have extended their hands to each itemsuch that the customer behavior index value is superimposed on the imageof the shelves 910. The customer behavior index value referred to hereis an index value that the index value calculator 293 calculates withregard to customers' behavior.

FIG. 4 is an explanatory diagram showing an example of display of thecustomer behavior index value by the display unit 220. In the exampleshown in FIG. 4, the display unit 220 displays an image of shelves 910in which items 920 are arranged in an area A11. The display unit 220displays images of the items in colors according to the number of timesthat customers have extended their hands to each of the items. Thedisplay unit 220 displays a legend indicating the association betweenthe number of times that customers have extended their hands to eachitem and the color in the area A12. The display unit 220 shows thenumber of times that customers have extended their hands to each item ina heat map format. The heat map referred to here is a display method inwhich the value of each element of a matrix is indicated by a color.

The number of times that customers have extended their hands to eachitem corresponds to an example of the customer behavior index value. Thedisplay unit 220 displays the customer behavior index value at theposition of the image of each item 920 in the image of the shelves 910with a color added to the image of the item. In this respect, thedisplay unit 220 displays the customer behavior index value togetherwith the image of the item.

However, the method in which the display unit 220 displays the customerbehavior index value is not limited to the method of adding a color tothe image of each item. For example, the display unit 220 may display acustomer behavior index value by adding a predetermined pattern such asthat of stripes or dots, instead of a color, to the image of each item.The display unit 220 may also display a number indicating the customerbehavior index value such that the number is superimposed on the imageof each item.

The display unit 220 also displays an index value, which indicates acorrelation between a behavior the customer has performed for an itemdesignated by a user of the image display system 1 and a behavior thatthe same customer has performed for an item other than the designateditem, at the position of an image of the item 920 in the image of theshelves 910. In the following, the user of the image display system 1 issimply referred to as a user.

FIG. 5 is an explanatory diagram showing an example of display, by thedisplay unit 220, of a customer behavior index value indicating acorrelation between a behavior that a customer has performed for an itemdesignated by the user of the image display system 1 and a behavior thatthe same customer has performed for an item other than the designateditem on the display unit 220. In the example of FIG. 5, the display unit220 displays an image of the shelves 910 in which items 920 are arrangedin an area A11, similar to the example of FIG. 4. The display unit 220also displays a legend indicating the association between the number oftimes that customers have extended their hands to each item and thecolor in an area A12.

In FIG. 5, the display unit 220 indicates items 920 designated(selected) by the user, unlike the case of FIG. 4. In the example ofFIG. 5, the user designates items 920 shown in the area A21 and thedisplay unit 220 indicates the designated items 920 by displayingstripes in the area A21. Hereinafter, the items 920 designated by theuser are referred to as designated items.

In the example of FIG. 5, the display unit 220 displays the count resultof the number of times that customers who extended their hands to thedesignated items (one or more times) have extended their hands to eachitem 920 other than the designated items (for example, a total count forall customers within a predetermined period) in a heat map format.Specifically, the display unit 220 displays an image of each item 920other than the designated items in a color corresponding to the numberof times that customers who extended their hands to the designated items(one or more times) have extended their hands to the item 920 other thanthe designated items. In the example of FIG. 5, customers who extendedtheir hands to the items (designated items) displayed in the area A21have extended their hands to an item displayed in an area A22 many timesand therefore the display unit 220 displays an image of the item 920 inthe area A22 in a color indicating that the number of times that thecustomers have extended their hands to the item is large.

For example, the operation input unit 230 includes input devices such asa keyboard and a mouse and receives user operations. The operation inputunit 230 corresponds to an example of the item designation receiver andreceives designation of an item. For example, the operation input unit230 receives designation of an item 920 through a user operation ofclicking the item 920 displayed on the display unit 220 with a mouse.

The storage unit 280 stores various types of information. The storageunit 280 stores sensing data obtained by the shelf-front behaviormeasurement sensor 110 or data that the controller 290 has obtained byprocessing the sensing data. For example, the storage unit 280 storesinformation that associates item identification information indicatingan item 920 to which a customer has extended their hand with the time atwhich the shelf-front behavior measurement sensor 110 has detected thehand of the customer (“association information” which will be describedlater). Further, the storage unit 280 stores the information thatassociates the item identification information indicating the item 920to which the customer has extended their hand with the time at which theshelf-front behavior measurement sensor 110 has detected the hand of thecustomer as information of a series of hand extensions of the samecustomer during a period from when the customer stops in front of theshelves to when leaving the shelves.

The item identification information referred to here is informationidentifying the item 920 placed on the shelves 910 for each of theitems. In the present embodiment, items of the same specification (forexample, items have the same name) are regarded as the same items.Predetermined information such as item names may be used as the itemidentification information or identification information dedicated tothe image display system 1 may be defined.

FIG. 6 is an explanatory diagram showing an exemplary data structure ofinformation in which a sensing time and item identification informationstored in the storage unit 280 are associated with each other(association information which will be described later). As shown inFIG. 6, the storage unit 280 stores a group number, a sensing time, anditem identification information in association with each other.

In the example of FIG. 6, the item identification information indicatesthe item 920, to which the customer has extended their hand, identifiedon the basis of the sensing data from the shelf-front behaviormeasurement sensor 110.

The sensing time indicates the time at which the shelf-front behaviormeasurement sensor 110 detected an object indicated in the sensing data.For example, the time at which the communicator 210 received the sensingdata may be used as the sensing time.

The group number is used as information for combining a series of handextensions detected from when the customer stops in front of the shelvesto when leaving the shelves into one group. On the basis of an imagecaptured by the shelf situation imaging device 120, the image displaydevice 200 detects a period from when the customer stops in front of theshelves to when leaving the shelves and adds the same group number toinformation (a combination of the sensing time and the itemidentification information) which is based on the sensing data that theshelf-front behavior measurement sensor 110 transmitted during thisperiod. This makes it possible to detect that one customer has extendedtheir hand to the shelves 910 a plurality of times as shown in theexample of FIG. 5.

However, the information stored in the storage unit 280 is not limitedto that shown in the example of FIG. 6 in which the sensing time and theitem identification information are associated with each other. Forexample, the storage unit 280 may store sensing data from theshelf-front behavior measurement sensor 110 and time informationindicating the sensing time in association with each other.Alternatively, the storage unit 280 may store information, which isobtained by converting the sensing data from the shelf-front behaviormeasurement sensor 110 into positions in the vertical and horizontaldirections of the shelves 910, in the form of coordinate values.Alternatively, the storage unit 280 may store information indicating,for each item 920, the number of times that the customer has extendedtheir hand to the item.

The storage unit 280 stores images captured by the shelf situationimaging device 120 in time series in association with the capturedtimes.

The storage unit 280 is realized using a storage device provided in theimage display device 200. The storage device provided in the imagedisplay device 200 may be a storage device built into the image displaydevice 200 or a storage device externally attached to the image displaydevice 200.

The controller 290 controls each part of the image display device 200 toperform various processes. The controller 290 is realized, for example,by a central processing unit (CPU) provided in the image display device200 reading and executing a program from the storage unit 280.

The approach situation information acquirer 291 acquires approachsituation information. The approach situation information referred tohere is information indicating whether or not at least part of thecustomer's body has approached the shelves 910 and indicating, when atleast part thereof has approached the shelves 910, a correspondingposition on the shelves 910 at each time.

Specifically, when the shelf-front behavior measurement sensor 110 hasdetected an object (in particular, a customer's hand), the approachsituation information acquirer 291 converts the sensing data intocoordinate values on the front face of the shelves 910.

For example, the storage unit 280 previously stores information whichindicates, for each shelf-front behavior measurement sensor 110, aposition in the horizontal direction of the shelves 910 at which theshelf-front behavior measurement sensor 110 detects an object. Inaccordance with a shelf-front behavior measurement sensor 110 that hasdetected an object, the approach situation information acquirer 291reads information regarding the position at which the shelf-frontbehavior measurement sensor 110 detects the object from the storage unit280.

The approach situation information acquirer 291 reads the distancebetween the object detected by the shelf-front behavior measurementsensor 110 and the shelf-front behavior measurement sensor 110 from thesensing data. The approach situation information acquirer 291 convertsthe read distance into a position in the vertical direction of theshelves 910.

The approach situation information acquirer 291 generates approachsituation information by associating reception time information of thesensing data, which is time information indicating the sensing time,with the corresponding coordinate value. The presence of the approachsituation information indicates that the shelf-front behaviormeasurement sensor 110 has detected the object.

When the shelf-front behavior measurement sensor 110 has detected anobject, it is assumed that at least part of the customer's body(especially the customer's hand) has approached the shelves 910. In thisrespect, the approach situation information indicates whether or not atleast part of the customer's body has approached the shelves 910 bypresence or absence of the information. The approach situationinformation indicates whether or not at least part of the customer'sbody has approached the shelves 910 and indicates, when at least partthereof has approached the shelves 910, a corresponding position on theshelves 910 by coordinate values. The position indicated by thecoordinates is a position on the shelves 910 at which at least part ofthe customer's body has approached the shelves 910.

The associator 292 associates the approach situation informationgenerated by the approach situation information acquirer 291 with one ofthe items 920 according to the position indicated by the approachsituation information. For example, the storage unit 280 previouslystores information that associates position information that theapproach situation information acquirer 291 detects from sensing datawith one of the items 920. Further, for each item 920 placed on theshelves 910, the storage unit 280 previously stores the position of theitem 920 on the shelves 910 as two-dimensional coordinate values in thehorizontal and vertical directions of the shelves 910. The associator292 reads the item 920 associated with the coordinate values indicatedby the approach situation information generated by the approachsituation information acquirer 291 from the storage unit 280, therebyassociating the approach situation information with the item 920.

The associator 292 generates information in which item identificationinformation identifying the associated item 920 and time informationindicating the sensing time are associated with each other and storesthe generated information in the storage unit 280. Hereinafter, theinformation generated by the associator 292, in which the itemidentification information and the time information indicating thesensing time are associated with each other is referred to asassociation information. The association information corresponds to anexample of the information indicating the item associated with theapproach situation information.

The index value calculator 293 calculates a customer behavior indexvalue on the basis of the association information generated by theassociator 292. For example, for each item 920, the index valuecalculator 293 counts the number of times that customers have extendedtheir hands to the item 920 within a predetermined period and calculatesa customer behavior index value indicating the count result as in theexample of FIG. 4.

Further, the index value calculator 293 calculates a customer behaviorindex value indicating a correlation between a behavior that a customerhas performed for a designated item and a behavior that the samecustomer has performed for an item other than the designated item. Forexample, as in the example of FIG. 5, for each item 920 other than thedesignated items, the index value calculator 293 calculates a customerbehavior index value indicating the number of times that customers whoextended their hands to the designated items have extended their handsto the item 920 other than the designated items.

The operation of the image display system 1 will now be described withreference to FIGS. 7 to 9.

FIG. 7 is a flowchart showing an example of a processing procedure inwhich the image display device 200 generates association information.When the image display device enters an operating state as the imagedisplay device 200 is connected to a power supply (is powered on), theimage display device 200 starts the processing of FIG. 7.

In the processing of FIG. 7, the approach situation information acquirer291 determines whether or not a customer's stopping in front of theshelves has been detected (step S101). Specifically, upon detectingpresence of a customer who has not been in front of the shelves untilnow in the image captured by the shelf situation imaging device 120, theapproach situation information acquirer 291 determines that thecustomer's stopping in front of the shelves has been detected. When aplurality of customers appeared in the image captured by the shelfsituation imaging device 120, the approach situation informationacquirer 291 may determine whether or not the customer's stopping hasbeen detected for each of the plurality of customers.

Upon determining that a customer's stopping in front of the shelves hasnot been detected (step S101: NO), the approach situation informationacquirer 291 returns to step S101. That is, the image display device 200waits for a customer to stop in front of the shelves.

Upon determining in step S101 that a customer's stopping in front of theshelves has been detected (step S101: YES), the approach situationinformation acquirer 291 determines a group number for the customer'sstopping (step S102). For example, an initial value of the group numbermay be set to 0 and the approach situation information acquirer 291 mayadd 1 to the group number each time a customer's stopping in front ofthe shelves is detected.

The approach situation information acquirer 291 determines whether ornot the customer's leaving the front of the shelves has been detected(step S103). More specifically, the approach situation informationacquirer 291 determines that the customer's leaving the front of theshelves has been detected upon detecting that the customer who has beenin front of the shelves is no longer present in the image captured bythe shelf situation imaging device 120. When a plurality of customersappeared in the image captured by the shelf situation imaging device120, the approach situation information acquirer 291 may determinewhether or not the customer's leaving has been detected for each of theplurality of customers.

Upon determining that the customer has left the shelf (step S103: YES),the approach situation information acquirer 291 returns to step S101.

Upon determining in step S103 that the customer has not left the shelves(step S103: NO), the approach situation information acquirer 291determines whether or not sensing data has been acquired from theshelf-front behavior measurement sensor 110 (step S104). Specifically,the approach situation information acquirer 291 determines whether ornot the communicator 210 has received sensing data.

Upon determining that sensing data has not been acquired (step S104:NO), the approach situation information acquirer 291 returns to stepS104. That is, the image display device 200 waits for sensing data fromthe shelf-front behavior measurement sensor 110.

Upon determining that sensing data has been acquired in step S104 (stepS104: YES), the approach situation information acquirer 291 generatesapproach situation information (step S105). Specifically, the approachsituation information acquirer 291 converts the sensing data received bythe communicator 210 into two-dimensional coordinate values on the frontface of the shelves 910 and generates approach situation information onthe basis of the obtained two-dimensional coordinate values.

The associator 292 generates association information on the basis of thetwo-dimensional coordinate values obtained in step S105 (step S106).Specifically, the associator 292 generates information that combines thegroup number determined in step S102, the sensing time, and the itemidentification information obtained from the two-dimensional coordinatevalues as association information for one row as in the example of thedata structure of FIG. 6.

The associator 292 causes the storage unit 280 to store the associationinformation (the information for one row) obtained in step S106 (stepS107).

After step S107, the associator 292 returns to step S103.

FIG. 8 is a flowchart showing a first example of a processing procedurein which the image display device 200 displays a customer behavior indexvalue. FIG. 8 shows an example where, for each item 920, the displayunit 220 displays the number of times that customers have extended theirhands to the item 920 in a color as described above with reference toFIG. 4. The image display device 200 starts the processing of FIG. 8,for example, when the operation input unit 230 has received a useroperation instructing display of a customer behavior index value.

In the processing of FIG. 8, for each item 920, the index valuecalculator 293 counts the number of times that customers have extendedtheir hands to the item 920 on the basis of the association informationobtained in the processing of FIG. 7 (step S201).

On the basis of the count result obtained in step S201, the index valuecalculator 293 adds colors to the images of the items 920 in the imageof the shelves 910 (step S202).

The index value calculator 293 causes the display unit 220 to displaythe image of the shelves 910 in which colors have been added to theitems 920 in step S202 (step S203).

After the step S203, the index value calculator 293 ends the processingof FIG. 8.

FIG. 9 is a flowchart showing a second example of the processingprocedure in which the image display device 200 displays a customerbehavior index value. FIG. 9 shows an example where, for each item 920other than a designated item, the display unit 220 displays the numberof times that customers who extended their hands to the designated itemhave extended their hands to the item 920 other than the designated itemin a color as described above with reference to FIG. 5. The imagedisplay device 200 starts the processing of FIG. 9 when the operationinput unit 230 has received a user operation instructing display of acustomer behavior index value indicating a correlation between items.

In the processing of FIG. 9, the operation input unit 230 receivesdesignation of an item 920 (step S301). For example, in a state wherethe display unit 220 displays an image of the shelves 910, the operationinput unit 230 receives an operation to designate one of the images ofthe items 920 included in the image of the shelves 910 with a mouseclick. The designation of the item 920 in step S301 is selection of thedesignated item.

The index value calculator 293 selects all pieces of associationinformation having the same group number as that of associationinformation indicating hand extension to the designated item from amongthe association information obtained in the processing of FIG. 7 (stepS302). That is, the index value calculator 293 acquires associationinformation indicating hand extension performed by customers whoextended their hands to the designated item.

For each item 920 other than the designated item, the index valuecalculator 293 counts the number of times that customers have extendedtheir hands to the item 920 other than the designated item with respectto the association information obtained in step S302 (step S303).

On the basis of the count result obtained in step S303, the index valuecalculator 293 adds colors to the images of the items 920 other than thedesignated item in the image of the shelves 910 (step S304).

The index value calculator 293 causes the display unit 220 to displaythe image of the shelves 910 in which colors have been added to theitems 920 in step S304 (step S305).

After the step S305, the index value calculator 293 ends the processingof FIG. 9.

As described above, the approach situation information acquirer 291acquires approach situation information indicating whether or not atleast part of the customer's body has approached the shelves 910 onwhich the items 920 are displayed and indicating, when at least partthereof has approached the shelves 910, a corresponding position on theshelves 910 at each time. The associator 292 associates the approachsituation information with one of the items 920 according to theposition indicated by the approach situation information. Thereby, theassociator 292 identifies an item 920 to which a customer has extendedtheir hand. The index value calculator 293 calculates an index valuerelating to the customer's behavior on the basis of informationindicating the item 920 associated with the approach situationinformation. The display unit 220 displays the index value calculated bythe index value calculator 293 at the position of the image of the item920 in the image of the shelves 910 on the basis of the association madeby the associator 292.

As a result, the image display system 1 can display informationindicating the customer's behavior in more detail. In particular, theimage display system 1 can display information indicating the customer'sbehavior for each item in more detail than that for each display shelf.By referring to this display, the user can perform more sophisticatedanalysis of customers' behavior.

The operation input unit 230 receives designation of an item by a useroperation. The index value calculator 293 calculates an index valueindicating the correlation between behaviors that the same customer hasperformed for a designated item and items other than the designateditem.

As a result, the display unit 220 can visually display the correlationbetween items. This allows the user to analyze the correlation betweenitems by referring to this display.

The user can determine to where on the shelves customers have extendedtheir hands using the image display system 1. This allows the user todetermine to where on the shelves customers can easily extend theirhands.

In addition, by using the image display system 1, the user can acquireinformation such as that regarding which items customers have extendedtheir hands to and that regarding which items customers have comparedwith each other. The user can use such information for marketingindividual items.

The display unit 220 may display the number of times that customers havepicked up each item in addition to or instead of displaying the numberof times that customers have extended their hands to each item describedabove with reference to FIGS. 4 and 5. For example, the image displaysystem 1 may include a camera that captures images from the top of theshelves 910 downward and the index value calculator 293 may analyze theimages of this camera to determine whether or not a customer has pickedup an item.

The display unit 220 may display information regarding whether or not anitem purchase has been made such as determination as to whether acustomer has purchased an item or returned the item to the shelves. Forexample, the index value calculator 293 compares an image captured bythe shelf situation imaging device 120 before a customer stops in frontof the shelves and an image captured by the shelf situation imagingdevice 120 after the customer has left the shelves. The index valuecalculator 293 determines that the customer has purchased the item upondetecting that the number of items has decreased. The index valuecalculator 293 determines that an item that the customer picked up hasreturned to the shelves upon detecting that the number of items has notdecreased.

The display unit 220 may display a period of time during which acustomer picked up an item. For example, the display unit 220 maydisplay an average period of time per pickup of an item by a customer,using each of a period of time from when the customer picks up the itemto when returning the item and a period of time from when the customerpicks up the item to when leaving the shelf as the period of time duringwhich the customer picked up the item.

As a result, the user can determine whether each individual item is onewhich the customer has purchased after careful consideration or onewhich the customer has decided in advance to purchase, and the user canuse such information for reference in marketing of items.

The image display system 1 may calculate a period of time during whichthe customer stays in front of the shelves and display the calculatedperiod of time together with the display of the shelf-front behaviorindex value described above.

The image display system 1 may include a camera for imaging a customerin front of the shelves from the side of the shelves and acquireinformation of at least one of the orientation of the face of thecustomer and the direction of the line of sight of the customer anddisplay the acquired information together with the display of theshelf-front behavior index value described above.

The image display system 1 may display information other than theshelf-front behavior information together with the display of theshelf-front behavior index value described above. For example, thestorage unit 280 may previously store information indicating whethereach customer is a member or not and the image display system 1 mayextract and display information such as information indicating thatpeople who are members are likely to pick up the item as the shelf-frontbehavior index value.

A minimal configuration of the present invention will be described belowwith reference to FIGS. 10 and 11.

FIG. 10 is an explanatory diagram showing a minimal configuration of theimage display device according to the present invention. The imagedisplay device 10 shown in FIG. 10 includes an approach situationinformation acquirer 11, an associator 12, an index value calculator 13and a display unit 14.

In this configuration, the approach situation information acquirer 11acquires approach situation information indicating approach of at leastpart of the customer's body to shelves on which items are displayed andindicating, when there is the approach, a position of the approach onthe shelves at each time. The associator 12 associates the approachsituation information with one of the items according to the positionindicated by the approach situation information. The index valuecalculator 13 calculates an index value relating to the customer'sbehavior on the basis of information indicating the item associated withthe approach situation information. The display unit 14 displays theindex value calculated by the index value calculator 13 at the positionof the image of the item in the image of the shelves on the basis of theassociation made by the associator 12.

As a result, the image display device 10 can display informationindicating the customer's behavior in more detail. In particular, theimage display device 10 can display information indicating thecustomer's behavior for each item in more detail than that for eachdisplay shelf. By referring to this display, the user can perform moresophisticated analysis of customers' behavior.

FIG. 11 is an explanatory diagram showing a minimal configuration of theimage display system according to the present invention. The imagedisplay system 20 shown in FIG. 11 includes a shelf-front behaviormeasurement sensor 21, an approach situation information acquirer 22, anassociator 23, an index value calculator 24 and a display unit 25.

In this configuration, the shelf-front behavior measurement sensor 21detects approach of at least part of the customer's body to a shelf onwhich items are displayed and generates sensing data indicating theposition of an approached portion on the shelves. On the basis of thesensing data, the approach situation information acquirer 22 generatesapproach situation information indicating whether or not there is theapproach and indicating, when there is the approach, the position on theshelves at each time. The associator 23 associates the approachsituation information with one of the items according to the positionindicated by the approach situation information. The index valuecalculator 24 calculates an index value relating to the customer'sbehavior on the basis of information indicating the item associated withthe approach situation information. The display unit 25 displays theindex value calculated by the index value calculator 24 at the positionof the image of the item in the image of the shelves on the basis of theassociation made by the associator 23.

As a result, the image display system 20 can display informationindicating the customer's behavior in more detail. In particular, theimage display system 20 can display information indicating thecustomer's behavior for each item in more detail than that for eachdisplay shelf. By referring to this display, the user can perform moresophisticated analysis of customers' behavior.

A program for realizing some or all of the functions of the controller290, the approach situation information acquirer 11, the associator 12,the index value calculator 13, the approach situation informationacquirer 22, the associator 23, and the index value calculator 24 may berecorded on a computer-readable recording medium, and processing of eachof the components may be performed by causing a computer system to readand execute the program recorded on the recording medium. The “computersystem” referred to here includes an OS and hardware such as peripheraldevices.

The “computer-readable recording medium” refers to a flexible disk, amagneto-optical disc, a ROM, a portable medium such as a CD-ROM, or astorage device such as a hard disk provided in the computer system.Further, the program described above may be one for realizing some ofthe functions described above and may also be one for realizing thefunctions described above in combination with a program already recordedon the computer system.

Although the embodiments of the present invention has been described indetail with reference to the drawings, the specific configuration is notlimited to the embodiments and includes design changes or the likewithout departing from the spirit of the present invention.

Priority is claimed on Japanese Patent Application No. 2016-057537,filed on Mar. 22, 2016, the content of which is incorporated herein byreference.

INDUSTRIAL APPLICABILITY

The present invention can be applied to an application in which it isindispensable to display information indicating a customer's behavior inmore detail.

REFERENCE SIGNS LIST

-   -   1, 20 Image display system    -   10, 200 Image display device    -   11, 22, 291 Approach situation information acquirer    -   12, 23, 292 Associator    -   13, 24, 293 Index value calculator    -   14, 25, 220 Display unit    -   21, 110 Shelf-front behavior measurement sensor    -   120 Shelf situation imaging device    -   210 Communicator    -   230 Operation input unit    -   280 Storage unit    -   290 Controller

The invention claimed is:
 1. An image display device comprising: atleast one memory configured to store instructions; and at least oneprocessor configured to execute the instructions to: determine adistance between a hand and a shelf; count a number of times a customermoved the hand toward a product displayed on the shelf based on thedetermined distance; generate a shelf image based on the number oftimes; and control a display to display the shelf image.
 2. The imagedisplay device according to claim 1, wherein the shelf image comprises asuperimposed heat map on an image of the shelf.
 3. The image displaydevice according to claim 1, wherein the shelf image includes aplurality of products including the product; and wherein the processorconfigured to execute the instructions to: generate the shelf image byadding a color corresponding to the number of times to an area of theshelf image including the product.
 4. The image display device accordingto claim 3, wherein the processor configured to execute the instructionsto: count a second number of times the customer moved the hand toward asecond product, of the plurality of products, displayed on the shelf;and generate the shelf image by adding a second color corresponding tothe second number of times to a second area of the shelf image includingthe second product.
 5. The image display device according to claim 3,wherein the shelf image includes a legend indicating an associationbetween the number of times and the color.
 6. The image display deviceaccording to claim 1, wherein the processor is further configured toexecute the instructions to: determine, by a sensor, a positon to whichthe customer moved the hand; and count the number of times the customermoved the hand toward the product based on the position.
 7. The imagedisplay device according to claim 1, wherein the processor is furtherconfigured to execute the instructions to: generate the shelf image asincluding a highlighting of the product, wherein the highlightingcomprises at least one of stripped and dotted lines superimposed overthe product.
 8. An image display method comprising: determining adistance between a hand and a shelf; counting a number of times acustomer moved the hand toward a product displayed on the shelf based onthe determined distance; generating a shelf image based on the number oftimes; and controlling a display to display the shelf image.
 9. Anon-transitory computer readable storage medium comprising instructionswhich when executed by a processor cause the processor to implement:determining a distance between a hand and a shelf; counting a number oftimes a customer moved the hand toward a product displayed on the shelfbased on the determined distance; generating a shelf image based on thenumber of times; and controlling a display to display the shelf image.